Gav's World

Setting Up Cluster Pi as a Load Balanced Web Server

There are a great many things that can be done with a cluster, even a low power one like a Raspberry Pi cluster where the worker nodes are only single core Pi Zeros. I have decided to play around with a reverse-proxy load balancing web server using Apache with various modules to accomplish the task.

Hardware

I've chosen to use a Pi 3 with 4x Zero Pis (no point in using ZeroW as they are connected in USB Gadget mode anyway, so have networking that way). If you use a Pi 4 as the controller, you can't reboot the controller without power cycling the worker nodes. Of course, you'll also need a Cluster HAT! I'd also recommend the Cluster HAT Case to house the cluster, although it really does need a bit more ventilation.

Operating Systems

Download and install all necessary images from ClusterCTRL... remember to add an empty "ssh" file in each boot partition to automatically enable SSH on all nodes. I picked the BRIDGE controller - primarily so that I can easily upload content to each worker from my Desktop PC.

Connect everything up and power up the cluster - only the controller Pi will power up, as the ClusterHAT needs to be instructed to apply power to all of the workers.

That's the easy bit! Now for just a few more steps...

File Systems

I have set up each worker node with its own microSD card for the operating system and also holding an entire copy of this website but with web stats on an NFS shared mount of the USB flash drive installed into the controller.

There are other ways that I could have set this up: for example I could have network booted each worker from the controller, but that would be slower to boot up, and would mean having to share the website storage... which yes, means only one copy on the cluster, but then each node has to access everything over its localised network which will be slower. Besides, the utility sitecopy makes it very simple to maintain the cluster's copies.

Instructions

1. ssh into the controller Pi and power up the workers:
   clusterctrl on
2. ssh into all nodes to apply all updates, change the default password and node names (if desired, using raspi-config). My nodes called:
   cluster-pi-c (controller)
   cluster-pi-w1 (worker 1)
   cluster-pi-w2 (worker 2)
   cluster-pi-w3 (worker 3)
   cluster-pi-w4 (worker 4)
3. On the controller Pi create an RSA key pair:
   ssh-keygen -b 2048
4. Then copy the public key to each of the workers, for example:
   ssh-copy-id pi@cluster-pi-w1
   (I also do that from my desktop PC to make remote login quicker)
5. Find the Universally Unique Identifier of the USB Flash drive so that we can automount it
   ls -l /dev/disk/by-uuid/
   And locate the UUID that links to "../../sda1"
6. Create a mount point for the USB Flash drive
   sudo mkdir /mnt/storage
7. Edit /etc/fstab to add the auto mount:
   sudo nano /etc/fstab
   And add the following (example UUID):
   # Automount the USB Flash drive
   UUID=12345-67890A /mnt/storage ext4 defaults 0 0
   Note that I have used ext4 on the USB drive - you may want vfat
8. Install nfs-kernel-server
   sudo apt-get install nfs-kernel-server
   Note that using the ClusterCTRL images, this is already installed.
9. Add an export point to /etc/exports
   sudo nano /etc/exports
   Note that using the ClusterCTRL images, there are a lot of exports already defined for cluster usage - I deleted them since I'm not using them.
10. Add the following line to make /mnt/storage read/write access to the whole sub-net serviced by the Pi:
    /mnt/storage cluster-pi-w*(rw,sync,no_subtree_check)
11. Restart the nfs-kernel-server
    sudo service nfs-kernel-server restart
12. On each worker:
    1. Create a mount point for the controller's storage
       sudo mkdir /mnt/cluster
    2. Edit /etc/fstab to add the auto mount:
       sudo nano /etc/fstab
       And add the following:
       # Mount the cluster storage
       cluster-pi-c:/mnt/storage /mnt/cluster nfs defaults 0 0
    3. Mount the controller's storage
       sudo mount /mnt/cluster
    4. Install Apache and PHP modules required for this website
       sudo apt -y install apache2 libapache2-mod-php php-cgi php-cli php-gd php-xml
    5. Add the user pi to the www-data group
       sudo usermod --append --groups www-data pi
       This just saves a lot of faffing around with file permissions, as I can FTP content as user pi.
    6. Change the ownership of /var/www/html to the www-data user/group and set the GUID bit (so that any users in group www-data, i.e. pi can write content)
       sudo chown www-data:www-data /var/www/html
       sudo chmod g+ws /var/www/html
    7. Use a web browser to test that the Apache installation is working on each of the worker nodes (since I've used the bridge controller, they are all accessible from the rest of the house network).
    8. Delete the default index.html ready for content upload
       sudo rm /var/www/html/index.html
    9. Unfortunately the combination of sitecopy and sftp (which is already active on the workers) doesn't like spaces in filenames, so we need an ftp daemon
       sudo apt -y install proftpd
13. At this point I can use sitecopy to upload content to all workers (having already defined my .sitecopyrc of course)
    sitecopy -ia
    sitecopy -ua
14. Now install apache on the controller
    sudo apt -y install apache2
15. Enable the required Apache modules
    sudo a2enmod proxy
    sudo a2enmod proxy_http
    sudo a2enmod proxy_balancer
    sudo a2enmod lbmethod_byrequests
16. Edit the default site configuration
    sudo nano /etc/apache2/sites-enabled/000-default.conf
    And add the following in between the <VirtualHost *:80> and </VirtualHost>tags:
    # Load Balancing for the Cluster
    ProxyRequests Off
    ProxyPreserveHost On
    
    <Proxy balancer://mycluster>
      BalancerMember "http://cluster-pi-w1:80"
      BalancerMember "http://cluster-pi-w2:80"
      BalancerMember "http://cluster-pi-w3:80"
      BalancerMember "http://cluster-pi-w4:80"
      ProxySet lbmethod=byrequests
    </Proxy>
    
    ProxyPass /balancer-manager !
    ProxyPass / balancer://mycluster/
    ProxyPassReverse / balancer://mycluster/
    
    <Location "/balancer-manager">
      SetHandler balancer-manager
      Require host localhost
      Require ip 127.0.0.1
      Require ip 192.168.1
    </Location>
    Then restart Apache
    sudo systemctl restart apache2
17. Ensure that the zeros power up after the controller has booted (the default is that they are off). Add the following to /etc/rc.local (before the exit 0 statement)
    # Power up the workers
    /sbin/clusterctrl on
18. Since I have my cluster powered via an Uninterruptible Power Supply (UPS) which is slaved from the server, I need a clean (and custom) shutdown script for the NUT monitor to call on a power failure. Create it as follows:
    sudo nano /sbin/cluster_poweroff
    And enter the following:
    #!/bin/bash
    ssh cluster-pi-w1 sudo poweroff
    ssh cluster-pi-w2 sudo poweroff
    ssh cluster-pi-w3 sudo poweroff
    ssh cluster-pi-w4 sudo poweroff
    
    sleep 60
    /sbin/clusterctrl off
    poweroff
    Make it executable:
    sudo chmod a+x /sbin/cluster_poweroff
19. Install nut-client and then configure as required - pointing the shutdown command to the custom command and ensuring that there is adequate time allocated.

The load balanced website is now up and running... with the balancer-manager active. To access it, simply add "balancer-manager" to the end of the URL, for example

Helpful Resources

• Apache Reverse Proxy Guide
• ClusterHAT Setup
• Clustered Apache 2.4 LoadBalancer displays no content on /balancer-manager
• Configure Apache load balancer with mod proxy